Inter-rater reliability of a composite health promotion scoring system developed in Singapore

INTRODUCTION@#In view of the important role of the environment in improving population health, implementation of health promotion programmes is recommended in living and working environments. Assessing the prevalence of such community health-promoting practices is important to identify gaps and make continuous and tangible improvements to health-promoting environments. We aimed to evaluate the inter-rater reliability of a composite scorecard used to assess the prevalence of community health-promoting practices in Singapore.@*METHODS@#Inter-rater reliability for the use of the composite health promotion scorecards was evaluated in eight residential zones in the western region of Singapore. The assessment involved three raters, and each zone was evaluated by two raters. Health-promoting practices in residential zones were assessed based on 44 measurable elements under five domains - community support and resources, healthy behaviours, chronic conditions, mental health and common medical emergencies - in the composite scorecard using weighted kappa. The strength of agreement was determined based on Landis and Koch's classification method.@*RESULTS@#A high degree of agreement (almost perfect-to-perfect) was observed between both raters for the measurable elements from most domains and subdomains. An exception was observed for the community support and resources domain, where there was a lower degree of agreement between the raters for a few elements.@*CONCLUSION@#The composite scorecard demonstrated a high degree of reliability and yielded similar scores for the same residential zone, even when used by different raters.

Geração de itens e validação de conteúdo de domínios e de itens para o indivíduo com osteoartrite do joelho: um estudo de método misto / Item generation and content validation of domains with item pool for the individual with knee osteoarthritis: a mixed-method study

INTRODUÇÃO: Existem várias escalas para avaliar as percepções subjetivas e os componentes individuais em indivíduos com osteoartrite (OA) de joelho. Até o momento, não há escalas disponíveis conhecidas para medir o equilíbrio combinado entre mobilidade, AVD e QV em OA de joelho com base na Classificação Internacional de Funcionalidade, Incapacidade e Saúde (CIF). OBJETIVO: Gerar itens e domínios relacionados aos problemas enfrentados pelos indivíduos com OA de joelhos e validar o conteúdo por especialistas. MÉTODOS: Os domínios e itens foram gerados através de pesquisa bibliográfica extensa (ELS) para extrair itens relacionados a equilíbrio, mobilidade, ADL e QV em indivíduos com OA em joelhos baseados na CIF e através de entrevista aprofundada direta (EAD) em 13 pessoas com OA de joelhos e três especialistas. A validação de conteúdo dos domínios e itens gerados foi validada por 10 especialistas por meio da pesquisa Delphi online. O índice mínimo de validação de conteúdo em nível de item (I-CVI) de 0,80 foi considerado para validar os itens identificados e o índice de validação de conteúdo em nível de escala geral (S-CVI) de 0,90 foi fixado para validar os itens gerados para uso no processo de desenvolvimento da escala. RESULTADOS: Os 117 itens gerados por EAD na ELS foram inicialmente agrupados em 18 domínios. A validação de conteúdo pelo método Delphi resultou em uma diminuição para 56 itens agrupados em 14 domínios com SCVI de 0, 93. CONCLUSÃO: O conjunto abrangente de itens de deficiência, limitação de atividade e restrição de participação para indivíduos com OA de joelhos nos domínios propostos foi desenvolvido e o conteúdo validado. Esses itens são recomendados para uso no desenvolvimento de uma nova escala abrangente de índice de osteoartrite do joelho (CKOAI).

INTRODUCTION: There are several scales to evaluate subjective perceptions and individual components in individuals with knee osteoarthritis (IKOA). Till date, no scale is available to measure the combined balance, mobility, ADL and QoL in IKOA based on the International Classification of Functioning, Disability and Health (ICF). OBJECTIVE: The purpose of the study was to generate items and domains related to problems faced by IKOA and to validate the content by experts. METHODS: The domains and items were generated through extensive literature search (ELS) to extract items related to symptoms, balance, mobility, ADL and QoL in IKOA based on the International Classification of Functioning, Disability and Health (ICF) and through in-depth direct interview (IDDI) from 13 IKOA and three experts. The content validation of domains and items generated were validated by 10 experts through online Delphi survey. Minimum itemlevel content validation index (I-CVI) of 0.80 was considered to validate the identified items and the overall scale-level content validation index (S-CVI) of 0.90 was fixed to validate the generated items to use in scale development process. RESULTS: 117 items generated by IDDI and ELS were grouped under 18 domains initially. Content validation by Delphi method resulted in reduction with 56 item pool being grouped under the 14 domains with SCVI is 0.93. CONCLUSION: The comprehensive impairment, activity limitation and participation restriction item pool for IKOA under the proposed domains, have been developed and content validated. These items are recommended for their use in development of new comprehensive knee osteoarthritis index scale (CKOAI).

Interrater reliability of fitness gram test to be used as a reliable tool for assessing physical fitness for school children in Guwahati urban society of North-East India

Background: The assessment of physical fitness using a specific tool has become an important part to find out the fitness level of children and adolescence. Fitness gram is a health-related fitness test that utilizes criterion-referenced standards on health-related components. Through the years research has shown that Fitness gram has become one of the most widely used programs in the United States, though it’s used in India is not popular.Methods: Seventy school students in the age group of 5 to 14 years were included in this study and randomly allocated for physical fitness test where 62 students have completed the test. Two physiotherapists as a rater were allotted and children were tested by them with Fitnessgram test battery in a gap of 1 week. Outcome measures includes Fitness gram test battery. Statistical analysis was done by Cronbach’s Alpha value has been computed for the Interrater reliability.Results: Sixty-two students with both gender (male 56.3% and female 43.7%, age -10.12±2.72) were assessed for physical fitness using Fitness gram test battery. All the test variables showed an excellent reliability (Cronbach’s alpha =0.91-0.95).Conclusions: The Fitness gram test battery has been found to be reliable tool and can be implemented for physical fitness test for the children.

Validity And Inter-Rater Reliability Of Postural Analysis Among New Raters

@#Work posture analysis is crucial in observing and reducing work-related musculoskeletal symptoms in the workplace. However, in a developing country, new raters are commonly assigned to conduct postural analysis to save on cost. This study aims to observe the validity and inter-rater reliability (defined as the degree of agreement among different raters) among new raters of three different commonly used work posture analysis methods: Rapid Upper Limb Assessment (RULA), Rapid Entire Body Assessment (REBA), and Ovako Workload Assessment System (OWAS). Fifty industrial engineering students, divided into five groups, who received prior training about the use of the methods, participated voluntarily in this study by observing ten different working postures in five different industries: the tofu, military equipment manufacturing, automotive maintenance and service, cracker, and milk-processing industries. One ergonomics expert also observed the working postures. Validity was observed based on the correlation between new raters’ ratings and the rating of the ergonomics expert. Inter-rater reliability within one group was calculated using the percentage of agreement and kappa value. The result shows high validity of RULA, REBA, and OWAS among new raters. There are insignificant differences in the inter-rater reliability of new raters among RULA, REBA, and OWAS. The implications of the result are discussed.

Concordancia entre observadores de videos de pacientes con sospecha diagnóstica de autismo y psicosis infantiles / Inter-rater Reliability in Videos of Patients with a Suspected Diagnostic of Autism and Child Psychosis

RESUMEN Introducción: Dados la complejidad del abordaje y el tratamiento del autismo y las psicosis infantiles y el impacto que tienen en la vida del niño y su familia, es fundamental ser cautos a la hora de hacer un diagnóstico que tiene implicaciones pronosticas. Se utiliza la perspectiva psicodinámica para acceder a la posibilidad de incluir en el diagnóstico aspectos tanto patológicos como sanos y promover el potencial evolutivo de cada paciente. Objetivo: Determinar la concordancia entre evaluadores y evaluar cuantitativa y cualitativa mente las variables que influyen en el diagnóstico de pacientes con sospecha de autismo y psicosis infantiles desde la perspectiva psicodinámica. Métodos: Se realizó un estudio de concordancia entre observadores basado en la evaluación diagnóstica de videos de pacientes con sospecha de autismo o psicosis infantiles que asisten a junta diagnóstica (sesión de observación) en el Instituto de Ortopedia Infantil Roosevelt. Resultados: Se obtuvieron valores de kappa que oscilaron en general entre 0,24 y 0,50, con una fuerza de concordancia que varió de leve a moderada y K = 0,24 para la organización de personalidad. Esta manera de diagnosticar toma en consideración los aspectos tanto patológicos como saludables que constituyen la organización de personalidad. Al final se discuten las limitaciones y los puntos que tener en cuenta en los estudios posteriores. Conclusiones: Se recomienda que evalúe al niño con perturbaciones mayores un equipo interdisciplinario, y no un solo observador, pues este espacio favorece la discusión, que es crucial a la hora de establecer un diagnóstico y evita lecturas parciales del funcionamiento psíquico del paciente que consulta.

ABSTRACT Introduction: Diagnosing and treating autism and child psychoses is very difficult; these pathologies impact not only the child's life but also the family as a whole. Therefore caution is required when giving a diagnosis with prognostic implications. We use the psychodynamic perspective in order to take into consideration both pathological and healthy diagnostic aspects, and to promote evolutionary potential of each patient. Objective: To determine the inter-rater reliability and to test quantitatively and qualitati vely the variables involved in the diagnoses of patients with suspected autism and child psychoses, based upon psychodynamic concepts. Methods: An inter-rater reliability study was carried on, based upon the diagnostic evaluation of videos of patients with suspected autism or child psychoses who attended the diagnostic meeting (observation session) at the Instituto de Ortopedia Infantil Roosevelt. Results: Kappa values were obtained, ranging from .24 to .50, with a reliability force varying from slight to moderate, and K=.24 for personality organization. This type of diagnosis takes into account both the pathological and healthy aspects which make up personality orga nization. Finally, limitations and aspects that should be considered in further studies were discussed. Conclusions: The results reinforce the need to evaluate a child with major disorders in an interdisciplinary team, and but a single observer, in order to allow discussion, and debate, and therefore, avoid partial readings of the patient's psychological functioning.

Confiabilidad interevaluador de la prueba video head impulse test en estudiantes de primer año de la generación 2016 de la Escuela de Fonoaudiología de la Universidad de Valparaíso Casa Central / Inter-rater reliability of the video head impulse test in students from the School of Speech and Language therapist of the University of Valparaiso main campus class 2016

RESUMEN Introducción: El video head impulse test es un método rápido, no invasivo y cómodo para evaluar el reflejo vestíbulo ocular, el cual está siendo utilizado con mayor frecuencia en la evaluación de pacientes con alteraciones del equilibrio. Sin embargo, en Chile se evidencia una falta de estudios relacionados a la confiabilidad interevaluador de dicha prueba. Objetivo: Determinar el nivel de concordancia de los resultados de la eficiencia del reflejo vestíbulo ocular de los canales semicirculares horizontales, obtenidos mediante la aplicación de la prueba video head impulse test, por parte de 3 evaluadores. Material y método: Estudio cuantitativo, correlacional, no experimental transversal descriptivo, con muestreo no probabilístico, de participación voluntaria. La prueba video head impulse test fue aplicada por 3 examinadores a 30 estudiantes que cumplieron con los criterios de inclusión. Resultados: El método de Bland y Altman mostró gran concordancia en las mediciones realizadas entre los 3 examinadores. La correlación intraclase para la ganancia del canal semicircular horizontal derecho e izquierdo fue de 0,85 y 0,91 respectivamente y de 0,75 para el valor de asimetría. Conclusión: El estudio mostró una confiabilidad interevaluador adecuada de la prueba video head impulse test para la evaluación de los canales semicirculares horizontales.

ABSTRACT Introduction: The video head impulse test is a rapid, non-invasive and comfortable method for the evaluation of the vestibulo-ocular reflex that is being used more often in the evaluation of patients with balance disorders. In Chile there is a lack of studies related to the inter-rater reliability for this test. Aim: To determine the level of agreement on the results for the vestibulo-ocular reflex of the horizontal semicircular canals obtained by the execution of the video head impulse test by 3 raters. Material and method: Quantitative, correlational, non-experimental, cross-sectional descriptive study with a type of non-probability sampling with voluntary participation. The video head impulse test was applied by 3 raters to 30 students who fulfilled the inclusion criteria. Results: The Bland-Altman method demonstrated good agreement in the measurements made between the 3 raters. The inter-rater reliability for the gain of the left and right horizontal semicircular canals was 0.85 and 0.91 respectively and 0.75 for the asymmetry value. Conclusion: The study showed an adequate inter-rater reliability for the video head impulse test in the evaluation of the horizontal semicircular canals.

Specific algorithm method of scoring the Clock Drawing Test applied in cognitively normal elderly / Método com algoritmo específico para pontuação do teste do desenho do relógio em idosos cognitivamente normais

ABSTRACT. The Clock Drawing Test (CDT) is an inexpensive, fast and easily administered measure of cognitive function, especially in the elderly. This instrument is a popular clinical tool widely used in screening for cognitive disorders and dementia. The CDT can be applied in different ways and scoring procedures also vary. OBJECTIVE: The aims of this study were to analyze the performance of elderly on the CDT and evaluate inter-rater reliability of the CDT scored by using a specific algorithm method adapted from Sunderland et al. (1989). METHODS: We analyzed the CDT of 100 cognitively normal elderly aged 60 years or older. The CDT ("free-drawn") and Mini-Mental State Examination (MMSE) were administered to all participants. Six independent examiners scored the CDT of 30 participants to evaluate inter-rater reliability. RESULTS AND CONCLUSION: A score of 5 on the proposed algorithm ("Numbers in reverse order or concentrated"), equivalent to 5 points on the original Sunderland scale, was the most frequent (53.5%). The CDT specific algorithm method used had high inter-rater reliability (p<0.01), and mean score ranged from 5.06 to 5.96. The high frequency of an overall score of 5 points may suggest the need to create more nuanced evaluation criteria, which are sensitive to differences in levels of impairment in visuoconstructive and executive abilities during aging.

RESUMO. O Teste do Desenho do Relógio (TDR) é uma barata e rápida medida de função cognitiva, de fácil aplicação, especialmente em idosos. Este instrumento é uma ferramenta clínica muito conhecida, amplamente utilizada no rastreamento de transtornos cognitivos e demência. O TDR pode ser aplicado de diferentes formas e a sua pontuação também varia. OBJETIVO: Os objetivos deste estudo foram analisar o desempenho dos idosos no TDR e avaliar a confiabilidade inter-examinadores do TDR pontuado por um método com algoritmo específico, adaptado a partir dos critérios estabelecidos por Sunderland et al. (1989). MÉTODOS: Analisamos o TDR de 100 idosos cognitivamente saudáveis com 60 anos de idade ou mais. O TDR ("desenho livre") e o Mini-Exame do Estado Mental (MEEM) foram administrados em todos os participantes. Seis avaliadores independentes pontuaram 30 TDR para avaliar a confiabilidade inter-examinadores. RESULTADOS E CONCLUSÃO: A pontuação 5 do algoritmo proposto ("Os números em ordem inversa ou concentrados") equivalente a 5 pontos na escala original de Sunderland foi a mais frequente (53,5%). O método com algoritmo específico do TDR utilizado teve alta confiabilidade entre avaliadores (p<0,01), e a média da pontuação variou entre 5,06 e 5,96. A alta frequência de 5 pontos na pontuação geral pode sugerir a necessidade da elaboração de critérios de avaliação mais sutis, que sejam sensíveis às diferenças entre indícios de comprometimento nas habilidades visuoconstrutivas e executivas durante o envelhecimento.

Inter-rater Reliability of Wisconsin Gait Scale and Gait Abnormality Rating Scale in Hemiplegic Patients after Stroke / 中国康复理论与实践

@#Objective To study the inter-rater reliability of Wisconsin Gait Scale (WGS) and Gait Abnormality Rating Scale (GARS) in patients with stroke. Methods 20 hemiplegic patients were required to walk on their comfortable speed and videotaped from frontal, backward and lateral. The video recordings were scored with WGS and GARS by 2 experienced physical therapists. Intraclass correlation coefficient (ICC) was calculated for the scores in each category and the total score. Results ICC for the WGS were 0.372~1, and were 0~0.875 for the GARS. Conclusion WGS is more appropriater to assess the gait of hemiplegic stroke patients than GARS.

Inter-rater Reliability of Wisconsin Gait Scale and Gait Abnormality Rating Scale in Hemiplegic Patients after Stroke / 中国康复理论与实践

Objective To study the inter-rater reliability of Wisconsin Gait Scale (WGS) and Gait Abnormality Rating Scale (GARS) in patients with stroke. Methods 20 hemiplegic patients were required to walk on their comfortable speed and videotaped from frontal, back-ward and lateral. The video recordings were scored with WGS and GARS by 2 experienced physical therapists. Intraclass correlation coeffi-cient (ICC) was calculated for the scores in each category and the total score. Results ICC for the WGS were 0.372~1, and were 0~0.875 for the GARS. Conclusion WGS is more appropriater to assess the gait of hemiplegic stroke patients than GARS.

Assessing Reliability of Medical Record Reviews for the Detection of Hospital Adverse Events / 예방의학회지

OBJECTIVES: The purpose of this study was to assess the inter-rater reliability and intra-rater reliability of medical record review for the detection of hospital adverse events. METHODS: We conducted two stages retrospective medical records review of a random sample of 96 patients from one acute-care general hospital. The first stage was an explicit patient record review by two nurses to detect the presence of 41 screening criteria (SC). The second stage was an implicit structured review by two physicians to identify the occurrence of adverse events from the positive cases on the SC. The inter-rater reliability of two nurses and that of two physicians were assessed. The intra-rater reliability was also evaluated by using test-retest method at approximately two weeks later. RESULTS: In 84.2% of the patient medical records, the nurses agreed as to the necessity for the second stage review (kappa, 0.68; 95% confidence interval [CI], 0.54 to 0.83). In 93.0% of the patient medical records screened by nurses, the physicians agreed about the absence or presence of adverse events (kappa, 0.71; 95% CI, 0.44 to 0.97). When assessing intra-rater reliability, the kappa indices of two nurses were 0.54 (95% CI, 0.31 to 0.77) and 0.67 (95% CI, 0.47 to 0.87), whereas those of two physicians were 0.87 (95% CI, 0.62 to 1.00) and 0.37 (95% CI, -0.16 to 0.89). CONCLUSIONS: In this study, the medical record review for detecting adverse events showed intermediate to good level of inter-rater and intra-rater reliability. Well organized training program for reviewers and clearly defining SC are required to get more reliable results in the hospital adverse event study.

Sensitivity, Specificity, Predictive Value and Inter-Rater Reliability of Malnutrition Screening Tools in Hospitalised Adult Patients

Nutrition screening is recommended as a first step of nutrition care to allow early identification and intervention of malnourished patients. The present study determined the validities and reliabilities of two malnutrition screening tools namely, the Malnutrition Universal Screening Tool (MUST) and Malnutrition Screening Tool (MST) among adult patients at the Hospital Kuala Lumpur. Methods: The sensitivity, specificity and predictive value of MUST and MST were conducted against the Subjective Global Assessment (SGA), anthropometric parameters including body mass index (BMI), calf circumference (CC),mid-upper arm circumference (MUAC) and energy intake. Inter-rater reliability was evaluated using kappa value (κ) to determine the level of agreement between raters. Results: A total of 151 subjects with mean age of 45.2 ± 13.7 years participated in this study. Prevalence of malnutrition according to MUST, MST and SGA was 34.4%, 33.8% and 19.9%, respectively. As compared to SGA, MUST and MST had a sensitivity of 96.6% and 93.3% respectively, whereas the specificity was 80.9% for both tools. The sensitivity and specificity of MUST against the anthropometric parameters (BMI, CC and MUAC) were between 53.8% to 88.8% and 67.4% to 69.9%, respectively. The sensitivity values for MST were between 46.1% to 63.6% and specificity values were between 64.4% to 67.6%. The inter-rater reliability of MUST was higher (substantial, mean (κ) = 0.78) than for MST (moderate, mean (κ) = 0.52). Conclusions: In conclusion, MUST was found to have similar validity levels but higher reliability result than MST. Based on our result, MUST is recommended for use in identifying adult patients who are at high risk of malnutrition. It can be used as a malnutrition screening tool but there is a need to evaluate the cost effectiveness of its implementation.

Validity of Workers’ Self-Reports. Evaluation of a Question Assessing Lifetime Exposure to Occupational Physical Activity.

Aims: In epidemiological studies exposure assessment based on questionnaires is the most cost-effective method. A question about lifetime exposure to occupational physical activity (OPA) was used in a population-based survey (part of the Copenhagen Aging and Midlife Biobank, CAMB). The aim of the study was to validate this question through a three-step process. Methodology: Firstly, the response process was studied by cognitive interviewing of 7 persons. Secondly, 64 persons participated in semi-structured interviews about their work-life, and expert judgments of exposure to OPA were compared with questionnaire-data. Exposure was 20 years of work in one of four categories of OPA: sedentary, standing and walking, moderate or high OPA. Kappa values were calculated for agreement and interpreted according to Landis and Koch’s criteria. Agreement was visualized in Bland-Altman plots. Thirdly, intra- and inter-rater reliability of expert judgments was tested. Results: Response process: The question had a complicated instruction, and the respondents found it hard to remember, categorize, and summate exposures. Semi-structured interviews: Kappa value for exposure to sedentary work was ‘substantial’ (0.71) but ‘fair’ for the other categories of OPA (0.27-0.29). Agreement between questionnaire and interview was higher in sedentary jobs and jobs with high OPA. Intra-rater reliability of expert judgments was ‘substantial’ or ‘moderate’ (0.60-0.71). Inter-rater reliability was high in sedentary jobs but lower in the more active jobs. Conclusion: Self-reports of lifetime exposure to sedentary work are valid in the CAMB cohort, whereas the validity of self-reports of exposure to high levels of occupational physical activity (OPA) are questionable. Thorough pre-testing of questions about lifetime OPA is recommended.

Inter-rater Reliability in a Clinical Performance Examination Using Multiple Standardized Patients for the Same Case / 한국의학교육

PURPOSE: The "standardization" of standardized patients (SP) is one of the most crucial factors for a successful clinical performance examination (CPX). This study aimed to examine the inter-rater reliability among SPs who portrayed the same case during a CPX. METHODS: The context was a CPX conducted under the supervision of CPX Seoul-Gyeonggi Consortium in K medical school in August 2007. K medical school ran 12 stations consisting of duplicated sets of 6 cases. In total, thirty SPs participated with 5 SPs acting each of the 6 cases. The SPs evaluated the student's performances in addition to portraying the cases. ANCOVA (analysis of covariance) was used to compare scores rated by the different SPs. The dependent variables were the case scores and the 4 subcomponent (history taking, physical examination, Clinical courtesy, and Patient-physician interaction) scores for each case; the independent variable was the SPs; and the covariate was the CPX total score. RESULTS: The Headache and Cough stations showed an acceptable level of reliability. Otherwise, Weight Loss and Facial Flushing failed to show consistent scores in all 4 subcomponents. Diarrhea and Lt. hemiparesis showed partial consistency. In terms of the subcomponents, the physical exam scores were most consistent and the patient-physician interaction scores were most inconsistent. CONCLUSION: This study tested the level of "standardization" of one set of CPX cases with mixed results. The authors hope that our results will contribute to quality assurance of CPX.

Management of esophagcal varices at Suratthani Hospital.

This is a retrospective study of 279 patients with bleeding esophageal varices in the medical department of Suratthani Hospital from 1 July 1996 to 30 April 2003. All patients were treated with endoscopic interventions (endoscopic sclerotherpy and/or rubber band ligation). The patients consisted of 67% men, and had an average of 54.9 years. Underlying diseases included liver cirrhosis (98.2%) and was alcoholic cirrhosis (59.5%), Child Pugh class B and C (77.7%), and EV grade III (62.1%). Clinical bleeding was considered moderate to severe upper GI hemorrhage in 64.4%) of patients. The patients were already in a high risk group, so mortality the rate was high (22.2%). Secondary prevention of recurrent bleeding with propanolol and endoscopic intervention should be widely used in general hospitals to decrease the rate of rebleeding and increase and increase early detection of esophageal varices in high risk patients.

The validity and reliability of clinical performance assessment of residents in the general surgery residency program.

A retrospective study of clinical performance assessment of residents in the general surgery residency program at the Department of Surgery,Faculty of Medicine Siriraj Hospital, during the 200-2001 and 2001-2002 academic years was done to evaluate the inter-rater reliability, internal structure, and ducational discriminant validity of the assessment. The inter-rater reliabilities were determined by using the intraclass correlation. Correlations between individual performance ratings were checked to determine how well faculty members differentiate their ratings with individual clinical skills. Finally, the percentage of marginal or unsatisfactory ratings was checked to demonstrate how sensitive the ratings were in identifying performance deficiencies among residents. From the evaluation of 42 residents in the 2000-2001 academic year, inter-rater reliability coefficients of individual ratings ranged from 0.06 to 0.93 with an average of 0.51. From the evaluation of 47 residents in the 2001-2002 academic year, inter-rater reliability coefficients ranged from 0.04 to 0.85 with an average of 0.49. Inter-rater reliabilities of performance ratings of third-year residents were at an acceptable standard for a medium-stake assessment. However, those of first and second-year residents should need some improvement. Tasks performed by first and second-year residents seem to have been inadequate for faculty members to evaluate their performance reliably. The correlation study between individual performance ratings demonstrated the lack of discrimination between many clinical skills in faculty ratings. Faculty members need some guidelines to differentiate between many items including relationship with patients, relationship with other doctors, and relationship with other workers; knowledge and judgment; work concentration and work effectiveness; and punctuality and responsibility. About ten percent of ratings were marginal or unsatisfactory which indicated that these ratings were sensitive enough to identify performance deficiencies among residents.